Complexity Measures from Interaction Structures 
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We evaluate information theoretic quantities that quantify complexity in terms of fc-th order 
statistical dependencies that cannot be reduced to interactions among k — 1 random variables. 
Using symbolic dynamics of coupled maps and cellular automata as model systems, we demonstrate 
that these measures are able to identify complex dynamical regimes. 
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I. INTRODUCTION 

In this paper we study a notion of complexity based 
on interaction among parts of a system. This is one of 
the first and most natural choices and has been consid- 
ered several times. Doing so, we will pursue a geometric 
approach going back to the works of Amari P, Q • In sta- 
tistical physics, for example, the complexity of a model is 
specified by the order of interaction. Firstly, one consid- 
ers the free theory with no interaction, and as the next 
approximation a theory of pair interaction. That means 
that there exists an energy function H, assigning to each 
state of the world x a real number H(x) such that the 
more probable states have a lower energy. If this func- 
tion has, using a suitable expansion into polynomials, 
only terms of order two, one speaks of pair interaction. 
The probability to find the world in the state x at tem- 
perature T is given by the Gibbs distribution 



P(x) 



-fm{x) 



where, k denoting Boltzmann's constant, /3 is the inverse 
temperature pp. By 



Z((3) 



we denoted the partition function, which is a normaliza- 
tion. 

In this work, we study a complexity measure that in- 
corporates the idea to quantify the amount of pairwise, 
triple wise,. . . -interaction in a rigorous way. We do so by 
using the exponential families of fc-interactions which will 
be defined in Section ITT] In this context, a fc-interaction 
is an interaction between k random variables. They can 
stand for k different particles in the spatial case, one 
variable in k time steps, or a mixture of both. Using the 
notion of distance from such an exponential family, we 
can quantify the amount of fc-wise interaction that can- 
not be explained by (k — l)-wise interaction. This allows 



us to quantify complexity. We explore this concept using 
two examples: Symbolic Dynamics of Coupled Map Lat- 
tices and Cellular Automata. For coupled map lattices 
we focus on spatial interaction. Still, the temporal and 
spatiotemporal features can be studied analogously as we 
will see for Cellular Automata. Our approach quantifies 
all interactions in a system. It does not necessarily re- 
fer to any neighborhood structure, any possible subsys- 
tem can contribute to this interaction. Allowing every 
subsystem, not necessarily local, to interact is motivated 
by possible applications as for example the identification 
of positions in the genome that carry a high informa- 
tion content. Interacting regions in the genome are not 
necessarily located spatially close to each other. For ex- 
ample the arrangement of the chromosome in space can 
bring sequentially distant sites into spatial proximity and 
thereby enable regulatory interactions. 

The paper is organized as follows. In Section fll] we 
introduce the necessary concepts from information the- 
ory and define our model systems. We give details on 
the models and numerical methods in Section IIIII and in 
Section [IV] we give the results. Section [V] contains con- 
cluding remarks. 



II. INFORMATION THEORY 

In this section we introduce exponential families of 
probability measures, the Kullback-Leibler divergence 
and other notions. Throughout the whole paper, the set 
of states of the system is of compositional structure. That 
means, for each site index v in a finite set V := {1, . . . , N} 
we have a finite configuration space X v . The set of all 
possible configurations is 



Xy 



'■— x X v , 



and likewise for each subset A C V, Xa ■= X vE aX v . 
Every real valued function on the set Xy can be seen as 
an element of the vector space 



p Ay 



v 



I}. 



As 
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Xy\ is finite, this space is isomorphic to 
Xy | . We will therefore call elements in 



r with 
vectors 



2 



or functions without preference. 

A. Families of Probability Measures 

We now study probability distributions on the set Xy . 
Naturally, these are also elements of M. Xv . Consider 



V{X V ) :={P& 



P(x) > 0, 



x<£X v 



P( X ) = 1 , 



the probability measures on the set Xy. Due to the 
compositional structure these probability measures are 
in fact only joint probabilities of a set of random vari- 
ables {X v : v € V}, where X v takes values in X v . V(Xy) 
has the geometrical structure of a (\Xy \ — l)-dimensional 
simplex. For P G V(Xy), we call 

supp(P) := {x e X v : P(x) > 0} 

the support of P. Then the probability distributions with 
full support are denoted 

V(X V ) := jp e V{X V ) : P(x) > 0,Va; £ Xy} . 



The notation is justified as V(Xy) is the closure of 

V(Xy). 

Example 1. We consider the example V = {1,2} in 
the binary case, i.e. X\ = X^ = {0, 1}. We have Xy — 
{0, l} 2 = {(00), (01), (10), ( 11)}, an d R Xv is just R 4 . The 
set of probability measures P{Xy) is a three-dimensional 
tetrahedron 

{(poo,Poi,Pio,Pn) G R> -Pn = 1 - Poo - Pox -Pxo) ■ 

It is the set of joint probabilities of two binary random 
variables. The extreme points of this tetrahedron are the 
4 unit vectors. 



For two distributions P,Q € V(Xy), we can now define 
a notion of distance by 



D(P || Q) := 



_ jE* e *^)l°g 2 S if (*) holds 



3C 



otherwise. 



Here, (*) is the condition supp(P) C supp(<5). D(P \\ 
Q) is called the Kullback-Leibler divergence or relative 
entropy. Although not a metric, it is non-negative and 
equals zero if and only if P = Q. 

In information geometry one studies families of prob- 
ability measures i.e. sub manifolds of V{Xy). A very 
natural class of such families arises if we consider the 
exponential map 



exp : 



V(Xy) f I * 



E 



x£X v 



It acts by component-wise exponentiation and normal- 
ization. 



Definition 2. Let X be a linear subspace of R v . We 
define the exponential family £j to be the image of X 
under the exponential map: 

Ex := exp (J). 

Exponential families are well known in statistical sci- 
ence. They have many nice properties with respect 
to maximum likelihood estimation and other inference 
methods. For our application we are, for a given P, in- 
terested in minimizing the distance D(P \\ Q) for Q in 
a given exponential family. It can be seen, that this is 
equivalent to maximum likelihood estimation of P in the 
family £. To do so, let P be the relative frequencies of 
an observation. Then the log-likelihood function with 
respect to Q is L(P, Q) := J2 x ex v P{x)logQ(x). Maxi- 
mum likelihood estimation is to find a Q that maximizes 
this function. On the other hand, if we add to — L the 
constant (with respect to Q) J2 x <ex v P( x ) P( x ) we 
find 

-L(P,Q)+ P(x)logP(x) 

xGX v 

P(x) 



]T P(x)\og 



xEX v 

= D(P || Q) 



Q{x) 



Therefore a maximizer of the likelihood is a minimizer of 
Kullback-Leibler distance. On the other hand, geomet- 
rically, minimization of D{P \\ Q) could be interpreted 
as a projection, since D is a kind of distance. This view 
is employed in information geometry. These projections 
are the key players in the definition of our complexity 
measure. 

In what follows, we define the concrete exponential 
families that we will use in the definition of our mea- 
sure. The idea that we want to follow was first explored 
by Darroch et al. 0. It is to consider exponential fami- 
lies of k interactions. Firstly, one defines the linear space 
of functions depending on only k of their arguments. Af- 
ter taking the exponential map of sums of such functions, 
we find probability distributions with only fc-interactions. 
For a given probability distribution, coming from a real 
system, we then want to quantify its complexity by mea- 
suring how far it is from being reducible to a theory of 
fc-interactions. 



B. Interaction Spaces 

We exploit the compositional structure of Xy to define 
the notion of fc-th order interactions. 

For any given A C V, we write x € Xy as x — 
{xa, Xy\ a) i.e. we distinguish between components in A 
and outside A. Then we define Z4 to be the subspace 
of functions that do not depend on the configurations 



3 



outside A: 

1a ■= j/ely: f(x A ,x v \ A ) = f(x A ,x' v y A ) 

for all x A e X A , x v \ A , x' v \ A G ^V\a} • 

Using these spaces as building blocks, one can define the 
interaction space corresponding to interactions between 
k arbitrary units 

l k := span X A , 

ACV, \A\=k 

i.e. just the span of all the vectors on the right hand 
side. Associated with each of the interaction spaces is an 
exponential family £ k '■— exp(Ifc). The interaction spaces 
are included in each other: Ii C I 2 C ■ • • C Xjv, therefore 
we have defined a hierarchy of exponential families: 



£ l c £ 2 c 



C £ 



N ■ 



This hierarchy was studied in [3, 0] . It has found var- 
ious applications in the theory of neural networks. The 
correspondence with the notions of statistical physics is 
as follows: A vector / G corresponds to an energy 
which has only k- interactions, but no higher interactions. 
It gives rise to a probability distribution 

P(x) = — e£ k . 

Vice versa every P in, for example, £2 has a (non-unique) 
representation as 



P(x) = [I 4> A (x) 

ACV:\A\=2 



exp 



. ACV:\A\=2 



Ia{xa) 



where the energy can be written with pair interactions 
only. 

Note that £n, corresponding to all functions, is V{Xy). 
In general, in the image of the exponential map, one has 
only distributions with full support. Probability zero cor- 
responds to infinite energy which is only achievable by 
limits of sequences of probability measures in an expo- 
nential family. It is an open question which of the possi- 
ble support sets are achievable by such limit probability 
measures in a given exponential family. This question is 
connected with the face structure of a convex polytope, 
the so called marginal polytope. See 0, Q for details. 
For our application one can just pass to the closure(in 
R") of the exponential family. In practice, if P is the 
distribution given by the relative frequencies observed 
in an experiment, we can compute the minimizer of the 
Kullback-Lcibler distance to an exponential family us- 
ing the iterative proportional fitting algorithm from Q. 
However, the complexity of the algorithm makes it un- 
feasible for values of \Xy\ S> 10 6 . 



Now we are about to introduce our complexity mea- 
sures. They are based on the notion of distance to an 
exponential family £ . We define 

D(P || £) := inf D{P \\ Q). 

Qee 

One can show that D(P \\ £) is continuous with respect 
to P G V{Xy). Given two exponential families £ C T on 
has 

D(P || £) - D{P || T) > 0. 

We now use differences like these as components of a 
vector valued complexity measure. 



C. Complexity measures 



Definition 



For a given P G 'P(Xy), we define a vector valued 
complexity measure I(P) := (I {1) {P), I {N) (P)) with 
components 

(P) := D(P || £ k -i) -D(P\\£ k ), k = 1, . . . , AT. 

In the next section we will argue for the choice and dis- 
cuss properties. 



2. Properties and Interpretation 

The main idea is that 1^ quantifies those dependen- 
cies between k nodes that are not captured by interac- 
tions between smaller subsets of nodes. However, this 
interpretation is a posteriori in the following sense: Con- 
sider an open system, such as an infinite CA. Although 
locally only pair interaction takes place, as the system 
evolves in time we find higher order correlations. When 
we speak of higher order interactions here, we mean an 
interpretation of higher correlations by a model of higher 
order interaction in a closed system. 

We look at examples for our quantities. Consider first 
a distribution P which factors over the units, i.e. it has 
a representation 



P(x) = Y[ P v (x v ), 



where P v are the marginals on single units. In this case P 
is an element of the closure £1 of the exponential family 
£1 and therefore I^(P) equals zero for k > 2. 

For higher interactions a similar statement is true. One 
has that P G £k if P admits a factorization as 



P(x) = 11 <j> A {x A ), 

A:\A\=k 



(1) 



while here the converse is not true. A given P G £ k \£k 
need not have this structure. In (|TJ) the functions (f>A have 



4 



the property to depend on their argument only through 
the part in A: xa- In this general case, the 4>a are not 
the marginals. If P G then = for I > k. 

Consider next a distribution in the synchronized case, 
i.e. from the value of one unit v all the other units are 
determined. Take as an example the distribution P with 
P(0 • • ■ 0) = P(l ■ ■ • 1) = | and P(x) = for all other 
elements. It can be seen that these distributions lie in 
the closure of £2, since, loosely speaking, by pair interac- 
tions this behavior can be explained. To do so let the 
second, third,. . . unit copy the value of the first unit, 
then the synchronized behavior is "constructed" using 
only conditions on pairs. Since P £ £2 we have that 
D(P || £ k ) = D(P || £ 2 ) = for all k > 2 and therefore 
the only non vanishing component of I(P) is . 

Another extreme case is given by the parity function. 
Let P be the uniform distribution on the following set of 
elements 

y := < x : Xv : X\ = xi mod 2 > . 

y is the set of all configurations with even parity. It can 
be seen that for this distribution P N ' is maximal while 
all other components vanish. This is due to the fact, 
that every choice of N — 1 units in is independent under 
P, while there is total (functional) dependence between 
all the N units. It can be seen that even the reverse 
statement is true. We elaborate a bit on this. Let g : 
{0, — > {0, 1} be a binary function on the N — 1- 
strings. Define the following set 

J', : {•<' -Vr : x (g(y),y) y e {0, l}^ 1 } 

This means that y g is formed by those strings where the 
nodes 2, . . . , N are chosen freely and the first node has 
the value of g. Let P be the distribution that is uniform 
on the set y g . Then it can be shown that if I N (P) > 
then either g is the parity function or 1 + g mod 2 is the 
parity function. 

The vector / has a natural correspondence to multi- 
information (or integration as it is termed in Q). Multi- 
information is a generalization of mutual information and 
is defined as 

M(P) :=Y / H l (P l )-H(P), 

where H{P) :— — J2 x ex P{x)\ogP(x) is the entropy 
of P and Pi is the {i}-marginal of P. Then Hi(Pi) := 
— ^2 xeX . Pi{x)\ogPi{x) is the marginal entropy in the 
unit i. Multi- information measures the distance of a dis- 
tribution from independence of the units. It allows a 
decomposition as 

N 

M{P) = D{P || £t) = Y, lik) ( p )- 

k=2 



See Q for these complexity measures, their properties, 
and the relation with TSE Complexity [|[. 

In it is shown that the maximizer of multi- 
information are contained in the closure of the exponen- 
tial family £2 and therefore have the /-vector concen- 
trated in j( 2 ' . 

In the following we compare our approach with the 
one of 10]. There the author distinguishes a chemical 
contrast, which corresponds to our JW. It quantifies the 
information that is contained in a sequence due to the av- 
erage density of zeros differing from |. Then, quantities 
k m , the correlational contrasts, were defined. These are 
in fact conditional mutual information, which measure 
the difference of the actual distribution to a Markov ap- 
proximation. They also appear in the effective measure 
complexity [l II ] which can be expressed as a weighted 
sum over these quantities. The relation to our measures 
is that k m = implies = 0. This can be understood 
from the fact that a distribution that has a Markov prop- 
erty with respect to m— 1 marginals will be representable 
by m — 1 interactions. On the other hand, a distribution 
that is representable by pair interactions does not neces- 
sarily admit a Markov representation as the example 

P(x) = — exp {c\X\X2 -I- 022:2X3 + C3X3X1} 

of a triangle with no three-way interaction shows. There- 
fore our measures give a finer view onto the interaction 
structure. 

Having identified "non complex" behaviors with ex- 
treme values of the components of / (P) we now look for 
dynamical systems where I is not concentrated in one 
component, because if several P > contribute, interac- 
tions on different scales coexist. This then could indicate 
complex behavior. We will next introduce our model sys- 
tems where we try to find complex behavior in the above 
sense. 

III. MODELS AND METHODS 
A. Model Systems 

1. Cellular Automata 

A possible natural choice for a model system are 
one dimensional (binary) cellular automata. They pro- 
duce discrete output and are therefore ideally suited for 
our method. Furthermore, CA were introduced by S. 
Wolfram [l2T] as paradigmatic models to study complex- 
ity. In this work, we restrict ourselves to some of the 
256 elementary binary CA studied by Wolfram [l3(. For 
these, starting from a configuration x^ S {0,1}^ the 
next configuration a;^' +1 - ) at a site i is computed as 

At+i) _ f (At) Jt) (t) \ 

for some function / : {0,1}^ -> {0,1}. Typically 
one chooses periodic boundary conditions, i.e. xn+i is 



5 



identified with x±. There exist exactly 256 functions 
{0, 1} — > {0, 1}, each characterized by its binary vector 
of values in {0, l} 8 . These vectors correspond to binary 
representations of integers smaller than 256 and the rules 
get numbered accordingly. 

In this simple model, only the next neighbors of node i 
influence this particular node. A priori, there are 88 ele- 
mentary CAs modulo the inversion and reflection symme- 
try. Furthermore, there are trivial cases like rule 0, etc. 
such that the number of "interesting" rules is smaller. 
In this paper, we pick out some specific rules that have 
found attention in the literature. Specifically, we study 
the rules 18, 20, 22, 30, 45, 50, 90, 110, 126 and 150. 

2. Coupled Map Lattices 

Our second model system: Symbolic Dynamics of Cou- 
pled Map Lattices. Let again V = {1, . . . , N} denote the 
set of node indexes. Assume we are given a connection 
structure between the nodes, specified by a graph G with 
vertices V(G) — V. By abuse of notation, the structure 
of the network is given by the symmetric adjacency ma- 
trix G = (gij)i,j=i,...,N G {0, l} jVxAr of the graph. We 
study the discrete time case t = 0, 1, 2, . . . and every node 
i G V carries a real value Xi(t) G [0, 1]. These values get 
updated simultaneously according to the so called cou- 
pled tent map rule [l4| 

Xi (t + 1) = -tojffait)) + (1 - e)f( Xi (t)), 

3 

where ki is the number of neighbors of node v and / : 
[0, 1] -> [0, 1] defined by 

t( \ \ 2x if0<a;< | 

X ^ f{x) = \2(l- X ) ifi<x<l ■ (2) 

is the tent map on the unit interval. Tent map lattices 
are a well established model to study synchronization and 
pattern formation in spatially extended systems [HI, [lj| . 

We will study binar y s ymbolic dynamics of coupled 
tent map network [l7l Il8|. For each node we consider 

Xt := {0, 1} and for every ACVwe have X A = {0, 1} |A| 
as in Section [Til Then the configuration space of the 
symbolic dynamics is Xy = {0, 1}^, the space of binary 
sequences of length N. To every real valued dynamics 
Xi(t) of node i, we consider the symbolic dynamics 

. v _ J if < Xi(t) < x* 
Sl[ ' _ \ 1 if x* < Xi (t) < 1 

where x* G [0, 1] is a given value. We obtain a global 
configuration of symbols s(t) = (si(t), . . . , Sn{£)) G Xy. 
Our aim is to utilize this symbolic time-series in order 
to evaluate the complexity of the full dynamics. Assum- 
ing that a dynamical system has arrived in a stationary 
state it will produce a stationary distribution on the set 



of symbolic strings. The method we propose is to capture 
this stationary distribution P and evaluate the complex- 
ity I(P). We will call a dynamics complex if it exhibits 
interactions of different orders at the same time. 



B. Simulation and Computation 

1. Cellular Automata 

As our computational power limits us to analyzing 
short sequences, we cut pieces of length 14 from CA of 
length 20000. These were iterated for 10 6 time steps, 
starting from an i.i.d. initial condition. To gain more in- 
sight in the structure of our measures we furthermore 
evaluate it for the "interaction cone" corresponding to 
a single time step. To do so we take all binary strings 
of length 3 and compute the value of the CA on these 
strings. We then append the output as the 4th digit to 
the input string and compute ■ ■ ■ ,1^ on the uni- 
form distribution on these strings. Of course in the run- 
ning CA one has to consider the influence of the station- 
ary distribution on the inputs. To measure this influence 
we also evaluate the elementary 4-strings when the input 
is sampled with the stationary distribution. 



2. Coupled Map Lattice 

To simulate the coupled map lattice one has to specify 
a graph. For our simulations, we use a fully connected 
graph ("globally coupled network"), and a circle graph 
("next neighbor coupling") of 10 nodes each. We initial- 
ize the networks by setting the Xi(0) to independently 
uniformly distributed random values in the real interval 
(0, 1). The dynamics generates two time-series, the real 
valued one x{t) — (&i(t), . . . ,xj^{t)) and, using x* = h, 
the symbolic one s(t) — (s\(t), . . . , s^r(t)). In the uncou- 
pled case, the value x* — \ gives the generating parti- 
tion. It makes the symbolic time series most informative 
about the true time series. We discard the symbols dur- 
ing a transient time, assuming that after the transient 
time (chosen here as 10 6 time steps), the system is in a 
stationary state. Treating the symbolic time-series as N 
realizations of a random variable with probability distri- 
bution P G ViXy) on the space of symbols we compute 
I(P) for this P. 

We also experimented with different values of the 
bound x* , but it turned out that this does not change 
the qualitative behavior. The value x* = | has received 
some attention as synchronization of the real valued dy- 
namics can be detected from the symbols [l9| . For our 
measures however, I does not seem to depend on the 
choice of a;* in such an essential manner. 
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FIG. 1: A snapshot of the highly regular binary symbolic dynamics of the circle graph at e = 0.476. Each row shows one node 
over time. White spots indicate the value zero, black squares one. 
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FIG. 2: A snapshot of the weakly coupled chaotic dynamic of the circle graph at e = 0.04. Each row shows one node over time. 
White spots indicate the value zero, black squares one. 



C. Computation of I (k) . 

As we have seen in Section III A[ computing an infor- 
mation projections, i.e. a minimizer of D(P \\ Q), for P 
given and Q G £ can be written as computing a max- 
imum likelihood estimation in some discrete statistical 
model. This estimation can be solved analytically only 
for some very simple models, such as E\. Nevertheless, for 
the general case good iterative algorithms exist. These 
go back to Kullback [2(| [Hj]. We have reimplemented 
the algorithm in parallel in the program CIPI which is 
freely available 22]. As the projections can be computed, 
the determination of 1^ is merely a subtraction of two 
KL-divergences. 



IV. RESULTS 
A. Cellular Automata 

In Table [J the experimental values for sequences of 
length 14, cut from large cellular automata are given. 
Then in Tables [TT] and IIIII the values of the /-Vector for 
elementary steps of each of the automata are printed. We 
distinguish between the uniform input and the true input 
statistic sampled from the stationary distribution of the 
respective CA. 

The values in Table U represent properties of the invari- 
ant measure of the respective CA. We confirm, that rules 
30,45, and 150 have the uniform distribution as their in- 
variant measure, therefore no interaction can be detected. 
The results for rule 22 show, that the long range corre- 
lations, known from [23|, [24j, can be detected using our 
measure. 

In the elementary interaction cone of 3 + 1 elements, 



rule 


j(D 


/(2) 


/(3) 


/ (4) 


J(5) 


J(6) 


J(7) 


J(8) 


18 


1.828 


2.237 


0.031 


0.010 


0.002 











20 


1.890 


1.556 


0.256 


0.028 


0.003 











22 


0.634 


0.479 


0.076 


0.179 


0.077 


0.035 


0.019 


0.006 


30 


























45 


























50 





9.011 




















54 


0.007 


2.658 


0.181 


1.409 


0.040 


0.004 








90 

















0.002 


0.002 


0.002 


110 


0.156 


3.738 


2.070 


0.035 


0.009 











126 





2.809 


0.307 


0.095 


0.004 











150 



























TABLE I: Results for Cellular Automata, stands for a value 
less than 10 -3 . 7 (9) to J (14) are not shown, as they vanish. 



rule 


7 (i) 


/(2) 


j(3) 


J (4) 


18 


0.189 


0.311 


0.500 





20 


0.189 


0.311 


0.500 





22 


0.046 


0.158 


0.800 





30 





0.189 


0.811 





45 





0.189 


0.811 





50 


0.046 


0.885 


0.069 





54 





0.189 


0.811 





90 








1 





110 


0.046 


0.158 


0.796 





126 


0.189 





0.811 





150 











1 



TABLE II: Cellular Automate 3 -> 1 statistics with uniform 
inputs. 
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0.8 
0.6 
0.4 
0.2 






- * - node 4 
— e— average 



CD 
T3 

o 
c 



2 
4 
6 
8 
10 



rv.i i ■ Viiii i r 

■ mm . i 1 '. iV 1 1 , "■■! ! !■£!■! ! ! I ' M l 1 '■■ . ■ il 



10 



20 



30 



40 



50 
t 



60 



70 



80 



90 



100 



FIG. 3: A snapshot of the complex dynamic of the circle graph at e = 0.482. In the top the activity of node 4 is plotted 
together with the average activity over all nodes. One can observe that for short periods of time synchronization effects occur, 
for instance between t = 30 and t = 40 node 4 is almost constant while the average also fluctuates less. Below the symbolic 
dynamics, with black as one, white as zero, of all 10 nodes is shown. The synchronization is visible here too as the pattern 
after t = 35 is very regular. 
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j(2) 


j(3) 


j(4) 


18 


0.757 


0.451 


0.635 





20 


0.822 


0.403 


0.535 





22 


0.260 


0.135 


0.908 





30 





0.189 


0.811 





45 





0.188 


0.812 





50 





3 








54 


0.003 


0.160 


0.847 





90 








1 





110 


0.059 


0.241 


0.804 





126 





0.500 


1 





150 











1 



TABLE III: Cellular Automate 3 
input statistics. 



1 statistics with specific 



we can observe the following principles: Rule 18 and rule 
20 build up correlations in the input which leads to over- 
all stronger correlations when the stationary distribution 
is used. One can also see, that the outputs are not uni- 
formly distributed leading to a high value of JW when 
the inputs are sampled from the stationary distribution. 
In the elementary time step of rule 22 a lot of triple wise 
interactions are detected. This property is also found 
when the inputs arc sampled from the stationary distri- 
bution. Rule 50 is a very simple rule which generates 



a periodic pattern of period 2. We see this represented 
here as (like in the synchronized case, described in the in- 
troduction) the /-vector is concentrated in I 2 . When the 
inputs are uniform on the other hand, this rule has some 
small correlations of three positions, these get eliminated 
as the system reaches the stationary distribution. Rule 
90 and rule 150 are additive rules that are modeled by 
the XOR function. Rule 150 takes XOR of every input, 
giving exactly the correlation between all 4 digits that 
was described in Section [Til Rule 90 is similar, it com- 
putes XOR of the inputs 1 and 3, leading to only triple 
wise interaction. 



For rules 22,30^45 there is evidence for the long range 
correlations in [25]. Here, we can distinguish rule 22 
from the other two, as rule 30 and rule 45 still show the 
uniform distribution in the outputs leading to j' 1 ) be- 
ing zero. Rule 30, Stephen Wolfram's "all time favorite 
rule" , is believed to have a high degree of randomness. 
(It might actually be used as a random number gener- 
ator in Mathematica). For us it is not distinguishable 
from rule 45. Rule 110, which is capable of universal 
computation shows no specific behavior with respect to 
our measure. Rule 126 has a mirror symmetry leading 
to being zero when the input is sampled uniformly. 
However, the stationary distribution induces a different 
symmetry. This leads to an interesting statistics, which 
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looks as follows 

f | for xG {(001), (Oil), (100), (110)} 
Pi2 6 (x) = h for x € {(000), (111)} 
[0 otherwise. 

It can be seen that the rule exactly computes XOR of 
input 1 and 3 on these configurations (which it does not 
do on all configurations). This leads to the contribution 
of I*- 3 -* . On the other hand, the nonexistence of the con- 
figurations (101) and (010) can only be accomplished by 
additional interactions of order 2. A comparison with 
rule 90 shows the non-additive nature of /. The exact- 
ness of the probabilities accounts for the exact values in 
I. In fact these values can be predicted from theory. 



network structures. Different snapshots of the symbolic 
dynamics of the circle graph are given in Figures [TJ [H 
and [3] 

For the fully connected graph, the nodes will be driven 
to synchronized chaos if the coupling strength is high 
enough [2(| [27j ■ We see this here as for e > 0.45 / is 
concentrated in I^ 2 \ It is interesting to observe that the 
indicate complex dynamics taking place on the edge 
of synchronized regimes. The largest Lyapunov exponent 
becomes smallest at e = 0.34 (but not zero or negative, 
which would indicate periodicity). The respective max- 
ima of the 1^ correspond to this. They are located at 
e 3 = 0.345, e 4 = 0.42, e 5 = 0.355, e 6 = 0.35. Also note 
that 1^ has at least a local maximum at the parameter 
value 0.35. 



B. Coupled Tent Maps 

1. Fully connected Graph 
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FIG. 4: Behavior of the largest Lyapunov exponent Ai for 
the fully connected graph(top), and the circle graph(bottom) . 
Note that the minima are attained in exactly the same pa- 
rameter region, in which the I^ k ' have maxima. 

In Figures [5] and [51 the values of to 1^ are plotted 
for the fully connected and the circle graph, depending 
on the coupling e. For k > 6, the 1^ are very small 
and depend strongly on the random initial condition, so 
that we don't evaluate them here. Figure 0] shows the 
behavior of the largest Lyapunov exponent for the two 



2. Circle Graph 

The circle graph of 10 nodes shows a specific 
"nearly periodic" behavior for parameter values e £ 
[0.463,0.481]. This is a specific feature due to the sym- 
metry of the system. It is not present for N ^ 10. We 
don't want to call the dynamics in that regime complex, 
as there is a high degree of regularity. In the respec- 
tive parameter region, the average activity taken over 
all nodes is almost constant. In the symbolic dynamics, 
2 of the nodes are constant themselves, 4 are periodic, 
while the remaining 4 are close to periodicity(see Fig- 
ure [T]) . Obviously, the complexity measures detect this 
phenomenon, as all I^ k \ k ^ 2 drop to zero. More inter- 
estingly, when the parameter e takes values close to where 
the regularity occurs we can observe very high values of 
the for k =/= 2. In this regime, the average activ- 
ity does not follow the one of the individual nodes, but 
fluctuates on the same scale, instead of averaging out. 
This indicates a complex dynamical structure driven by 
higher order correlations among the nodes. See Figure [3] 
for a snapshot of the dynamics. On the onset of the syn- 
chronization one can additionally observe "part time syn- 
chronization" . This means that a nearly periodic state, 
as described above, emerges, but is not stable. After a 
couple of time steps (depending on the exact parameter 
value), it dissolves again. 

To understand the reason for the peak close to the pe- 
riodic phenomenon better we investigated the transition 
region closer. First, we took a very long sequence that 
shows transitions. Then we separated the two phases, 
saving them to different files and analyzed them sepa- 
rately. The result here is, that the unordered state shows 
the same / vector as the region left of the peak, while the 
periodic sequences of course have I concentrated in 1^ , 
as the theory predicts. If the two types of sequences 
are mixed then higher order correlations appear, leading 
to the peak. This corresponds to the more general and 
unsolved problem whether the complexity of a convex 
combination of two distributions is related to the com- 
plexities of the individual constituents. 
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FIG. 5: The Values of /' fc ' for the fully connected graph. For e > 0.45, the network is in the state of synchronized chaos. This 
is fully captured by pair interactions, therefore is maximal, while the 1^ for k > 2 vanish. On the edge to synchronized 
chaos high correlations on different scales are detected. 



CONCLUSION 



We have introduced complexity measures, suggested 
by [3, H|, that quantify the interactions of k parts of a 
system that cannot be explained by interaction between 



fewer parts of that system. Then, after studying elemen- 
tary theoretical properties and their interpretation, we 
have explored the measures in numerical studies. Using 
the elementary cellular automata one can already study 
the behavior of the measures in special cases. There are 
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FIG. 6: The values of 1^ for the circle graph. The inlays show the parameter region e £ [0.46, 0.49]. 



specific effects, such as emergence of higher correlations, 
that can be found with our method. Commonly studied 
quantities like correlation functions capture only part of 
what we called I^ 2 \ Our measures therefore give a more 
global view. 

Symbolic dynamics of coupled map lattices are far less 
understood from a theoretical point of view. We have 



shown that using the symbolic dynamics one can detect 
important dynamical properties of the underlying real 
valued dynamics. This is true despite the fact that only 
spatial correlations are used in the computation of the 
j( k ) _ We stress the point that our measures take as input 
only the statistics of the system and make no model as- 
sumptions. They are suitable for exploratory analysis, as 
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it is done in sequence based genetics. In an exploratory 
study one has no order parameters, as these quantities 
need to be constructed from some model first. 

Interestingly we observed that complex behavior takes 
place on the edge of synchronization, similar to the com- 
mon wisdom of "complex dynamics on the edge of chaos" . 

plays a special role in this context. As synchroniza- 
tion phenomena are entirely captured by pair interac- 
tions, we would suggest to measure complexity in terms 
of for k > 2. There is more work to do, investigating 
this notion of complexity in different kinds of models. 

Nevertheless the universality of the results remains to 



be investigated, as our system size is very small and 
the computation of the gets infeasible as the sys- 
tem grows. Therefore, for the time being our approach 
is practically limited, while still conceptually appealing. 
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